Amortizing replication log updates for transactions

ABSTRACT

Updates for transactions to a replication log for a database may be amortized. As updates are received as part of an active transaction, replication log records may be generated. When the size of replication log records exceeds a transaction size threshold, the replication log records may be stored in a remote data store instead of a local data store. When a request to commit the active transaction is received, an replication log for the database may be updated to include the replication log records from a transaction data object in the remote data store.

BACKGROUND

Database replication techniques offer users the opportunity to replicatechanges made to one database across different locations, systems,services or devices, providing, among other features wide availabilityand accessibility of data stored in a database. Database replicationtechniques, however, are not implemented without costs. In order toensure consistency with replicas of the database features, such asreplication logging may be implemented in addition to the features thatsupport the source database workload to perform, among many otheroperations, transactions or other updates to the source database.Therefore, techniques that can reduce the cost of database replicationtechniques while providing similar consistency assurances are highlydesirable.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a logical block diagram illustrating amortizing replicationlog updates for transactions, according to some embodiments.

FIG. 2 is a logical block diagram illustrating a provider network thatimplements a database service and separate storage service thatimplements amortizing replication log updates for transactions,according to some embodiments.

FIG. 3 is a logical block diagram illustrating various components of adatabase service and separate storage service, according to someembodiments.

FIG. 4 is a logical block diagram illustrating amortization ofreplication logs to remote storage, according to some embodiments.

FIG. 5 is a logical block diagram illustrating a data flow for bothamortizing and non-amortizing replication log records for transactions,according to some embodiments.

FIG. 6 is a high-level flow chart illustrating methods and techniquesfor amortizing replication log updates for transactions, according tosome embodiments.

FIG. 7 is a high-level flow chart illustrating methods and techniquesfor updating a replication log, according to some embodiments.

FIG. 8 is a block diagram illustrating a computer system that mayimplement at least a portion of systems described herein, according tosome embodiments.

While embodiments are described herein by way of example for severalembodiments and illustrative drawings, those skilled in the art willrecognize that the embodiments are not limited to the embodiments ordrawings described. It should be understood, that the drawings anddetailed description thereto are not intended to limit embodiments tothe particular form disclosed, but on the contrary, the intention is tocover all modifications, equivalents and alternatives falling within thespirit and scope as defined by the appended claims. The headings usedherein are for organizational purposes only and are not meant to be usedto limit the scope of the description or the claims. As used throughoutthis application, the word “may” is used in a permissive sense (i.e.,meaning having the potential to), rather than the mandatory sense (i.e.,meaning must). The words “include,” “including,” and “includes” indicateopen-ended relationships and therefore mean including, but not limitedto. Similarly, the words “have,” “having,” and “has” also indicateopen-ended relationships, and thus mean having, but not limited to. Theterms “first,” “second,” “third,” and so forth as used herein are usedas labels for nouns that they precede, and do not imply any type ofordering (e.g., spatial, temporal, logical, etc.) unless such anordering is otherwise explicitly indicated.

“Based On.” As used herein, this term is used to describe one or morefactors that affect a determination. This term does not forecloseadditional factors that may affect a determination. That is, adetermination may be solely based on those factors or based, at least inpart, on those factors. Consider the phrase “determine A based on B.”While B may be a factor that affects the determination of A, such aphrase does not foreclose the determination of A from also being basedon C. In other instances, A may be determined based solely on B.

The scope of the present disclosure includes any feature or combinationof features disclosed herein (either explicitly or implicitly), or anygeneralization thereof, whether or not it mitigates any or all of theproblems addressed herein. Accordingly, new claims may be formulatedduring prosecution of this application (or an application claimingpriority thereto) to any such combination of features. In particular,with reference to the appended claims, features from dependent claimsmay be combined with those of the independent claims and features fromrespective independent claims may be combined in any appropriate mannerand not merely in the specific combinations enumerated in the appendedclaims.

DETAILED DESCRIPTION

Various techniques for amortizing replication log updates fortransactions are described. Replication logs for databases, such asMySQL's Binlog feature, may support various types of applications thatutilize additional copies of a database, such as additional read copies,Change Data Capture, and database migration. When replication logging isenabled, a database may copy all transaction data into the replicationlog on commit of a transaction, in various embodiments, which can take along time if the transaction has large amount of data. In someembodiments, such as network-based (e.g., cloud-based) applications,where the data storage destination of a replication log is not local,the copy time may become extended even further. If the database were tofail (e.g., application crash, power outage, etc.) when the copying oftransaction data to the replication occurs, the recovery time can belengthy since the database may need to rollback changes in both thedatabase itself (e.g., using undo logs of the transaction) and restorethe replication log to a consistent state. This downtime cansignificantly impact the performance of client applications of adatabase, because recovery operations may cause the database to beoffline or otherwise unavailable.

Techniques for amortizing replication log updates for transactions maysignificantly reduce the risk of long recovery operations, such as inscenarios where a very large transaction would be vulnerable to failureduring the updating of replication log to commit the transaction.Instead, amortizing replication log updates for transactions canminimize the amount of work remaining to commit a transaction. In thisway, the window of time during which database failures can causesignificant down time are reduced significantly.

For example, without the implementation of amortizing replication logupdates for transactions, downtime to recover a very large transactionis sometimes several hours or days, effectively shutting down clientapplications of the database during that recovery. With amortizingreplication log updates for transactions, that downtime can be reducedto minutes (or less), in some scenarios. Additionally, a database canhandle much larger transactions because utilization of local storagecapacity to store replication log records can be minimized. The databasecan also experience improvements in throughput for very largetransactions, since the commit time can be significantly reduced.Moreover, the reduction in commit time also provides other transactionsand workloads with more resources to complete, such as by not blockingsmaller transactions on commit (e.g., by reducing the time holding lockfor serializing replication log writes).

FIG. 1 is a logical block diagram illustrating amortizing replicationlog updates for transactions, according to some embodiments. Twodifferent timelines are illustrated. Non-amortized transaction timeline102 may illustrate, in various embodiments, a replication log techniquethat does not amortize the transmission of replication log records forinclusion in a replication log. Consider transaction A. Transaction Amay be an active transaction for which replication log records 110 arebeing generated as updates to the database as part of the transactionare received. When commit request 106 for transaction A is received, logrecords for transaction A may be stored 120 to a data store for thereplication log, which may be a non-local or remote data store (e.g.,over a network connection, such as the transmission of replication logrecords discussed below with regard to FIGS. 2-4 ). The period of timefor transmitting and updating the replication log to include thetransaction after commit may be indicated by non-amortized failurewindow 140 (which may represent a time during which a failure couldcause a significant downtime to recover the database).

Amortized transaction timeline 104 may illustrate a replication logtechnique that amortizes transmission of replication log records forinclusion in the replication log. In various embodiments, replicationlog records 110 may be stored locally (e.g., in cache and/or localpersistent storage in some embodiments as discussed below in FIGS. 2-7 )until a transaction size threshold 108 is reached. After exceeding thetransaction size threshold 108 (e.g., according to a size of storagespace, such as 512 MB, or number of records), replication log recordsmay be stored to a remote data store 130 for inclusion in thereplication log. When commit request 106 is received, the replicationlog may be updated to include the log records, as indicated at 132.However, the amortized failure window 150 is significantly smaller thanthe non-amortized failure window 140. Moreover, additional optimizationsmay be implemented to update the replication log 132, such as updatingthe replication log without additional data copying from a transactiondata object that stores the data log, updating without duplicatingstructural and accuracy features for the replication log, such as thecalculation of record offsets and checksums, and inserting the logrecords into a location in the replication log consistent with the orderapplied by the replication log, such as an order where transactions arestored in the order in which they committed to the database, in someembodiments.

In some cases, non-amortized replication techniques may be more orequally performant. For example, for transactions less than thetransaction size threshold, the update replication log operations maytake more time and/or other processing costs than performing at one timeafter commit. Thus, for some transactions, as discussed below withregard to FIG. 6 , non-amortized replication log techniques may beperformed concurrently with amortized replication log techniques.

Please note, FIG. 1 is provided as a logical illustration and is notintended to be limiting as to the physical arrangement, size, or numberof components, modules, or devices to implement such features.

The specification first describes an example of a provider network thatmay implement a database service and storage service, according tovarious embodiments. Included in the description of the examplenetwork-based services to amortize replication log updates fortransactions. The specification then describes a flowchart of variousembodiments of methods for amortizing replication log updates fortransactions. Next, the specification describes an example system thatmay implement the disclosed techniques. Various examples are providedthroughout the specification.

FIG. 2 is a logical block diagram illustrating a provider network thatimplements a database service and separate storage service thatimplements amortizing replication log updates for transactions,according to some embodiments. Provider network 200 may be set up by anentity such as a company or a public sector organization to provide oneor more services (such as various types of cloud-based computing orstorage) accessible via the Internet and/or other networks to clients250. Provider network 200 may include numerous data centers hostingvarious resource pools, such as collections of physical and/orvirtualized computer servers, storage devices, networking equipment andthe like (e.g., computing system 1000 described below with regard toFIG. 8 ), needed to implement and distribute the infrastructure andservices offered by the provider network 200.

In some embodiments, provider network 200 may implement variousnetwork-based services, including database service(s) 210, a storageservice(s) 220, and/or one or more other virtual computing services 240(which may include various other types of storage, processing, analysis,communication, event handling, visualization, and security services).Database service(s) 210 may implement various types of database systemsand formats (e.g., relational, non-relational, graph, document, timeseries, etc.) and the respective types of query engines to performqueries to those databases. Storage service(s) 220 may include manydifferent types of data stores, including a log-structured storageservice or other storage services as discussed below with regard toFIGS. 3 and 4 , in some embodiments and may store both database data 222and replication data 224.

Clients 250 may access these various services offered by providernetwork 200 via network 260. Likewise network-based services maythemselves communicate and/or make use of one another to providedifferent services. For example, storage service 220 may store data 222for databases managed by database service 210, in some embodiments. Itis noted that where one or more instances of a given component mayexist, reference to that component herein may be made in either thesingular or the plural. However, usage of either form is not intended topreclude the other

In various embodiments, the components illustrated in FIG. 2 may beimplemented directly within computer hardware, as instructions directlyor indirectly executable by computer hardware (e.g., a microprocessor orcomputer system), or using a combination of these techniques. Forexample, the components of FIG. 2 may be implemented by a system thatincludes a number of computing nodes (or simply, nodes), each of whichmay be similar to the computer system embodiment illustrated in FIG. 8and described below. In various embodiments, the functionality of agiven service system component (e.g., a component of the databaseservice or a component of the storage service) may be implemented by aparticular node or may be distributed across several nodes. In someembodiments, a given node may implement the functionality of more thanone service system component (e.g., more than one database servicesystem component).

Generally speaking, clients 250 may encompass any type of clientconfigurable to submit network-based services requests to network-basedservices platform 200 via network 260, including requests for databaseservices (e.g., a request to execute a transaction or query with respectto a database, a request to manage a database, such as a request toenable or disable performing queries across different types of queryengines, etc.). For example, a given client 250 may include a suitableversion of a web browser, or may include a plug-in module or other typeof code module that can execute as an extension to or within anexecution environment provided by a web browser. Alternatively, a client250 (e.g., a database service client) may encompass an application, aweb server, a media application, an office application or any otherapplication that may make use of provider network 200 to store and/oraccess one or more databases. In some embodiments, such an applicationmay include sufficient protocol support (e.g., for a suitable version ofHypertext Transfer Protocol (HTTP)) for generating and processingnetwork-based services requests without necessarily implementing fullbrowser support for all types of network-based data. That is, client 250may be an application that can interact directly with network-basedservices platform 200. In some embodiments, client 250 may generatenetwork-based services requests according to a Representational StateTransfer (REST)-style network-based services architecture, a document-or message-based network-based services architecture, or anothersuitable network-based services architecture. In some embodiments, aclient of database service(s) 210 may be implemented within providernetwork 200 (e.g., on another service 240, such as virtual computingservice).

In some embodiments, a client 250 (e.g., a database service client) mayprovide access to a database hosted in database service 210 to otherapplications in a manner that is transparent to those applications. Forexample, client 250 may integrate with an operating system or filesystem to provide storage in accordance with a suitable variant of thestorage models described herein. However, the operating system or filesystem may present a different storage interface to applications, suchas a conventional file system hierarchy of files, directories and/orfolders, in one embodiment. In such an embodiment, applications may notneed to be modified to make use of the storage system service model.Instead, the details of interfacing to provider network 200 may becoordinated by client 250 and the operating system or file system onbehalf of applications executing within the operating systemenvironment.

Client(s) 250 may convey network-based services requests (e.g., arequest to query a database or perform a transaction at a database) toand receive responses from services implemented as part of providernetwork 200 via network 260, in some embodiments. In variousembodiments, network 260 may encompass any suitable combination ofnetworking hardware and protocols necessary to establishnetwork-based-based communications between clients 250 and providernetwork 200. For example, network 260 may generally encompass thevarious telecommunications networks and service providers thatcollectively implement the Internet. Network 260 may also includeprivate networks such as local area networks (LANs) or wide areanetworks (WANs) as well as public or private wireless networks. Forexample, both a given client 250 and provider network 200 may berespectively provisioned within enterprises having their own internalnetworks. In such an embodiment, network 260 may include the hardware(e.g., modems, routers, switches, load balancers, proxy servers, etc.)and software (e.g., protocol stacks, accounting software,firewall/security software, etc.) necessary to establish a networkinglink between given client 250 and the Internet as well as between theInternet and provider network 200. It is noted that in some embodiments,clients 250 may communicate with provider network 200 using a privatenetwork rather than the public Internet. For example, clients 250 may beprovisioned within the same enterprise as a database service system(e.g., a system that implements database service 210 and/or storageservice 220). In such a case, clients 250 may communicate with providernetwork 200 entirely through a private network 260 (e.g., a LAN or WANthat may use Internet-based communication protocols but which is notpublicly accessible).

Services within provider network 200 (or provider network 200 itself)may implement one or more service endpoints to receive and processnetwork-based services requests, such as requests to access data pages(or records thereof), in various embodiments. For example, providernetwork 200 services may include hardware and/or software to implement aparticular endpoint, such that an HTTP-based network-based servicesrequest directed to that endpoint is properly received and processed, inone embodiment. In one embodiment, provider network 200 services may beimplemented as a server system to receive network-based servicesrequests from clients 250 and to forward them to components of a systemwithin database service 210, storage service 220 and/or another virtualcomputing service 240 for processing.

In some embodiments, provider network 200 (or the services of providernetwork 200 individually) may implement various user managementfeatures. For example, provider network 200 may coordinate the meteringand accounting of user usage of network-based services, includingstorage resources, such as by tracking the identities of requestingclients 250, the number and/or frequency of client requests, the size ofdata tables (or records thereof) stored or retrieved on behalf of user,overall storage bandwidth used by users or clients 250, class of storagerequested by users or clients 250, or any other measurable user orclient usage parameter, in one embodiment. In one embodiment, providernetwork 200 may also implement financial accounting and billing systems,or may maintain a database of usage data that may be queried andprocessed by external systems for reporting and billing of client usageactivity. In some embodiments, provider network 200 may be to collect,monitor and/or aggregate a variety of storage service system operationalmetrics, such as metrics reflecting the rates and types of requestsreceived from clients 250, bandwidth utilized by such requests, systemprocessing latency for such requests, system component utilization(e.g., network bandwidth and/or storage utilization within the storageservice system), rates and types of errors resulting from requests,characteristics of stored and requested data pages or records thereof(e.g., size, data type, etc.), or any other suitable metrics. In someembodiments such metrics may be used by system administrators to tuneand maintain system components, while in other embodiments such metrics(or relevant portions of such metrics) may be exposed to clients 250 toenable such clients to monitor their usage of database service 210,storage service 220 and/or another virtual computing service 230 (or theunderlying systems that implement those services).

In some embodiments, provider network 200 may also implement userauthentication and access control procedures. For example, for a givennetwork-based services request to access a particular database, providernetwork 200 may implement administrative or request processingcomponents that may ascertain whether the client 250 associated with therequest is authorized to access the particular database. Providernetwork 200 may determine such authorization by, for example, evaluatingan identity, password or other credential against credentials associatedwith the particular database, or evaluating the requested access to theparticular database against an access control list for the particulardatabase. For example, if a client 250 does not have sufficientcredentials to access the particular database, provider network 200 mayreject the corresponding network-based services request, for example byreturning a response to the requesting client 250 indicating an errorcondition, in one embodiment. Various access control policies may bestored as records or lists of access control information by databaseservice 210, storage service 220 and/or other virtual computing services230, in one embodiment.

FIG. 3 is a logical block diagram illustrating various components of adatabase service and separate storage service, according to someembodiments. Database service 210 may implement one or more differenttypes of database systems with respective types of query engines foraccessing database data as part of the database. In the example databasesystem implemented as part of database service 210, a database enginehead node 310 may be implemented for each of several databases and alog-structured storage service 350 (which may or may not be visible tothe clients of the database system). Clients of a database may access adatabase head node 310 (which may be implemented in or representative ofa database instance) via network utilizing various database accessprotocols (e.g., Java Database Connectivity (JDBC) or Open DatabaseConnectivity (ODBC)). However, log-structured storage service 350, whichmay be employed by the database system to store data pages of one ormore databases (and redo log records and/or other metadata associatedtherewith) on behalf of clients, and to perform other functions of thedatabase system as described herein, may or may not benetwork-addressable and accessible to database clients directly, indifferent embodiments. For example, in some embodiments, log-structuredstorage service 350 may perform various storage, access, change logging,recovery, log record manipulation, and/or space management operations ina manner that is invisible to clients of a database engine head node310.

As previously noted, a database instance may include a single databaseengine head node 310 that implements a query engine 320 that receivesrequests, like request 312, which may include queries or other requestssuch as updates, deletions, etc., from various client programs (e.g.,applications) and/or subscribers (users), then parses them, optimizesthem, and develops a plan to carry out the associated databaseoperation(s). Query engine 320 may return a response 314 to the request(e.g., results to a query) to a database client, which may include writeacknowledgements, requested data pages (or portions thereof), errormessages, and or other responses, as appropriate. As illustrated in thisexample, database engine head node 310 may also include a storageservice engine 330 (or client-side driver), which may route readrequests and/or redo log records to various storage nodes withinlog-structured storage service 350, receive write acknowledgements fromlog-structured storage service 350, receive requested data pages fromlog-structured storage service 350, and/or return data pages, errormessages, or other responses to query engine 320 (which may, in turn,return them to a database client).

In this example, query engine 320 or another database system managementcomponent implemented at database engine head node 310 (not illustrated)may manage a data page cache, in which data pages that were recentlyaccessed may be temporarily held. Query engine 320 may be responsiblefor providing transactionality and consistency in the database instanceof which database engine head node 310 is a component. For example, thiscomponent may be responsible for ensuring the Atomicity, Consistency,and Isolation properties of the database instance and the transactionsthat are directed that the database instance, such as determining aconsistent view of the database applicable for a query, applying undolog records to generate prior versions of tuples of a database. Queryengine 320 may manage an undo log to track the status of varioustransactions and roll back any locally cached results of transactionsthat do not commit.

FIG. 3 illustrates various interactions to perform various requests,like request 312. For example, a request 312 that includes a request towrite to a page may be parsed and optimized to generate one or morewrite record requests 321, which may be sent to storage service engine330 for subsequent routing to log-structured storage service 350. Inthis example, storage service engine 330 may generate one or more redolog records 335 corresponding to each write record request 321, and maysend them to specific ones of the storage nodes 360 of log-structuredstorage service 350. Log-structured storage service 350 may return acorresponding write acknowledgement 337 for each redo log record 335 (orbatch of redo log records) to database engine head node 310(specifically to storage service engine 330). Storage service engine 330may pass these write acknowledgements to query engine 320 (as writeresponses 323), which may then send corresponding responses (e.g., writeacknowledgements) to one or more clients as a response 314.

In another example, a request that is a query may cause data pages to beread and returned to query engine 320 for evaluation and processing or arequest to perform query processing at log-structured storage service350 may be performed. For example, a query could cause one or more readrecord requests 325, which may be sent to storage service engine 330 forsubsequent routing to log-structured storage service 350. In thisexample, storage service engine 330 may send these requests to specificones of the storage nodes 360 of log-structured storage service 350, andlog-structured storage service 350 may return the requested data pages339 to database engine head node 310 (specifically to storage serviceengine 330). Storage service engine 330 may send the returned data pagesto query engine 320 as return data records 327, and query engine maythen evaluate the content of the data pages in order to determine orgenerate a result of a query sent as a response 314. As discussed belowwith regard to FIG. 4 , some requests to store replication log records331 may be performed as part of performing replication log techniques(e.g., to amortize the transmission of replication records to areplication log).

In some embodiments, various error and/or data loss messages 341 may besent from log-structured storage service 350 to database engine headnode 310 (specifically to storage service engine 330). These messagesmay be passed from storage service engine 330 to query engine 320 aserror and/or loss reporting messages 329, and then to one or moreclients as a response 314.

In some embodiments, the APIs 331-341 of log-structured storage service350 and the APIs 321-329 of storage service engine 330 may expose thefunctionality of the log-structured storage service 350 to databaseengine head node 310 as if database engine head node 310 were a clientof log-structured storage service 350. For example, database engine headnode 310 (through storage service engine 330) may write redo log recordsor request data pages through these APIs to perform (or facilitate theperformance of) various operations of the database system implemented bythe combination of database engine head node 310 and log-structuredstorage service 350 (e.g., storage, access, change logging, recovery,and/or space management operations).

Note that in various embodiments, the API calls and responses betweendatabase engine head node 310 and log-structured storage service 350(e.g., APIs 321-329) and/or the API calls and responses between storageservice engine 330 and query engine 320 (e.g., APIs 331-341) in FIG. 3may be performed over a secure proxy connection (e.g., one managed by agateway control plane), or may be performed over the public network or,alternatively, over a private channel such as a virtual private network(VPN) connection. These and other APIs to and/or between components ofthe database systems described herein may be implemented according todifferent technologies, including, but not limited to, Simple ObjectAccess Protocol (SOAP) technology and Representational state transfer(REST) technology. For example, these APIs may be, but are notnecessarily, implemented as SOAP APIs or RESTful APIs. SOAP is aprotocol for exchanging information in the context of Web-basedservices. REST is an architectural style for distributed hypermediasystems. A RESTful API (which may also be referred to as a RESTful webservice) is a web service API implemented using HTTP and RESTtechnology. The APIs described herein may in some embodiments be wrappedwith client libraries in various languages, including, but not limitedto, C, C++, Java, C# and Perl to support integration with databaseengine head node 310 and/or log-structured storage service 350.

In some embodiments, database data for a database of database service210 may be organized in various logical volumes, segments, and pages forstorage on one or more storage nodes 360 of log-structured storageservice 350. For example, in some embodiments, each database may berepresented by a logical volume, and each logical volume may besegmented over a collection of storage nodes 360. Each segment, whichlives on a particular one of the storage nodes, may contain a set ofcontiguous block addresses, in some embodiments. In some embodiments,each segment may store a collection of one or more data pages and achange log (also referred to as a redo log) (e.g., a log of redo logrecords) for each data page that it stores. Storage nodes 360 mayreceive redo log records and to coalesce them to create new versions ofthe corresponding data pages and/or additional or replacement logrecords (e.g., lazily and/or in response to a request for a data page ora database crash). In some embodiments, data pages and/or change logsmay be mirrored across multiple storage nodes, according to a variableconfiguration (which may be specified by the client on whose behalf thedatabases is being maintained in the database system). For example, indifferent embodiments, one, two, or three copies of the data or changelogs may be stored in each of one, two, or three different availabilityzones or regions, according to a default configuration, anapplication-specific durability preference, or a client-specifieddurability preference.

In some embodiments, a volume may be a logical concept representing ahighly durable unit of storage that a user/client/application of thestorage system understands. A volume may be a distributed store thatappears to the user/client/application as a single consistent orderedlog of write operations to various user pages of a database, in someembodiments. Each write operation may be encoded in a log record (e.g.,a redo log record), which may represent a logical, ordered mutation tothe contents of a single user page within the volume, in someembodiments. Each log record may include a unique identifier (e.g., aLogical Sequence Number (LSN)), in some embodiments. Each log record maybe persisted to one or more synchronous segments in the distributedstore that form a Protection Group (PG), to provide high durability andavailability for the log record, in some embodiments. A volume mayprovide an LSN-type read/write interface for a variable-size contiguousrange of bytes, in some embodiments.

In some embodiments, a volume may consist of multiple extents, each madedurable through a protection group. In such embodiments, a volume mayrepresent a unit of storage composed of a mutable contiguous sequence ofvolume extents. Reads and writes that are directed to a volume may bemapped into corresponding reads and writes to the constituent volumeextents. In some embodiments, the size of a volume may be changed byadding or removing volume extents from the end of the volume.

In some embodiments, a segment may be a limited-durability unit ofstorage assigned to a single storage node. A segment may provide alimited best-effort durability (e.g., a persistent, but non-redundantsingle point of failure that is a storage node) for a specificfixed-size byte range of data, in some embodiments. This data may insome cases be a mirror of user-addressable data, or it may be otherdata, such as volume metadata or erasure coded bits, in variousembodiments. A given segment may live on exactly one storage node, insome embodiments. Within a storage node, multiple segments may live oneach storage device (e.g., an SSD), and each segment may be restrictedto one SSD (e.g., a segment may not span across multiple SSDs), in someembodiments. In some embodiments, a segment may not be required tooccupy a contiguous region on an SSD; rather there may be an allocationmap in each SSD describing the areas that are owned by each of thesegments. As noted above, a protection group may consist of multiplesegments spread across multiple storage nodes, in some embodiments. Insome embodiments, a segment may provide an LSN-type read/write interfacefor a fixed-size contiguous range of bytes (where the size is defined atcreation). In some embodiments, each segment may be identified by asegment UUID (e.g., a universally unique identifier of the segment).

In some embodiments, a page may be a block of storage, generally offixed size. In some embodiments, each page may be a block of storage(e.g., of virtual memory, disk, or other physical memory) of a sizedefined by the operating system, and may also be referred to herein bythe term “data block”. A page may be a set of contiguous sectors, insome embodiments. A page may serve as the unit of allocation in storagedevices, as well as the unit in log pages for which there is a headerand metadata, in some embodiments. In some embodiments, the term “page”or “storage page” may be a similar block of a size defined by thedatabase configuration, which may typically a multiple of 2, such as4096, 8192, 16384, or 32768 bytes.

As discussed above, log-structured storage service 350 may perform somedatabase system responsibilities, such as the updating of data pages fora database, and in some instances perform some query processing on data.As illustrated in FIG. 3 , storage node(s) 360 may implement data pagerequest processing 361, replication log processing 363, and datamanagement 365 to implement various ones of these features with regardto the data pages 367 and page log 369 of redo log records among otherdatabase data in a database volume stored in log-structured storageservice. For example, data management 365 may perform at least a portionof any or all of the following operations: replication (locally, e.g.,within the storage node), coalescing of redo logs to generate datapages, snapshots (e.g., creating, restoration, deletion, etc.), logmanagement (e.g., manipulating log records), crash recovery, and/orspace management (e.g., for a segment). Each storage node may also havemultiple attached storage devices (e.g., SSDs) on which data blocks maybe stored on behalf of clients (e.g., users, client applications, and/ordatabase service subscribers), in some embodiments. Data page requestprocessing 361 may handle requests to return data pages of records froma database volume, and may perform operations to coalesce redo logrecords or otherwise generate a data pages to be returned responsive toa request. Replication log processing 363 may handle requests to storereplication logs to transaction objects and update replication logsstored in or associated with logical replication log 371.

In at least some embodiments, storage nodes 360 may provide multi-tenantstorage so that data stored in part or all of one storage device may bestored for a different database, database user, account, or entity thandata stored on the same storage device (or other storage devices)attached to the same storage node. Various access controls and securitymechanisms may be implemented, in some embodiments, to ensure that datais not accessed at a storage node except for authorized requests (e.g.,for users authorized to access the database, owners of the database,etc.).

FIG. 4 is a logical block diagram illustrating amortization ofreplication logs to remote storage, according to some embodiments.Database engine head node 410 may implement replication log manager 420which may perform various techniques to implement amortized andnon-amortized replication logging techniques. Transaction updates 402may be received at replication log record generator 450 which maygenerate a log record that describes or indicates the update to thedatabase as well as the associated transaction. In some embodiments, thereplication log may be a logical replication log (e.g., describing thechanges so that the changes can be performed). In other embodiments, thereplication log may be a physical replication log (e.g., including a newrecord, field, item, or other value that stores the value after theupdate is applied). Replication log record generator 450 may supportdifferent types or formats of replication log records, which may beselectable in response to a request, in some embodiments.

Replication log records 452 may be provided to replication log manager420 which may determine where to store and when to move replication logrecords 452, for instance according to the techniques discussed belowwith regard to FIGS. 5 and 6 . Replication log manager 420 may update421 local memory 430 to store replication log records in thecorresponding transaction cache 432. Replication log manager 420 mayupdate 423 local persistent storage 440 to store log records in localtransaction data objects 442, in some embodiments, which are not beingamortized (or not yet being amortized). Replication log manager 420 maystore replication log records 425 in transaction data objects 462 foramortized replication techniques and directly in replication log 464when a commit is received for a non-amortized transaction. Note that insome embodiments, transaction data objects 462 and 442 may beappend-only data objects.

Replication log manager 420 may perform update operations 427 to includetransactions in replication log 464 in response to a commit. In someembodiments, updates to replication log 464 may be synchronizedaccording to various synchronization features, such as latches. Forexample, replication log manager 420 may update a replication log indexto indicate the inclusion of additional replication log records, renamea transaction data object 462 for inclusion in replication log 464,update pointers (e.g., from other log records in replication log 464and/or a transaction data object 462) to point to the replicationrecords of the transaction data object 462 in order to include thereplication log records of the transaction data object 462 withoutrewriting the replication log records into the replication log.Replication log manager 420 may also perform recovery operations 429 toreplication log in the event of a failure that causes a roll-back oftransactions to make replication log 464 consistent with the database,in some embodiments, as discussed above with regard to FIG. 1 . Forinstance, replication log manager 420 may determine whether to applyundo records to rollback a transaction that did not commit in both thereplication log and at the database.

FIG. 5 is a logical block diagram illustrating a data flow for bothamortizing and non-amortizing replication log records for transactions,according to some embodiments. Different transactions may be handleddifferent, in various embodiments. For example, transaction A andtransaction B may be handled with a non-amortized replication technique,as discussed below with regard to FIGS. 6 and 7 . For example,transaction A non-amortized data stream 510 may have replication recordsstored in transaction A cache 512 until a cache size threshold isexceeded. Then, replication records may be spilled or otherwise storedto transaction A local object 514. Both transaction A cache 512 andtransaction A local object 514 may be in local storage 560. Similarly,transaction B non-amortized data stream 520 may have replication recordsstored in transaction B cache 522 until a cache size threshold isexceeded. Then, replication records may be spilled or otherwise storedto transaction B local object 524. Both transaction B cache 522 andtransaction B local object 524 may be in local storage 560. A request tocommit these transactions as a group may be received, as indicated at532 (e.g., by a query engine or by replication manager to updatereplication log 550 as a batch) the records may be ordered according tothe transaction order of replication log 550 stored directly intoreplication log 550.

Other non-amortized transactions may be committed individually. Forexample, transaction C non-amortized data stream 540 may havereplication records stored in transaction C cache 542 until a cache sizethreshold is exceeded. Then, replication records may be spilled orotherwise stored to transaction C local object 544. Both transaction Ccache 542 and transaction C local object 544 may be in local storage560. When a single or individual commit 536 is received, then therecords from transaction C local data object 544 may be stored directlyin replication 550 according to the transaction order for replicationlog 550.

For an amortized transaction, like transaction N, an amortized datastream 530 may be first stored into transaction N cache 532, when atransaction size threshold is exceeded, then the replication records maybe spilled or otherwise stored into transaction N remote data object 534in remote storage 570. Note that this happens before a request to commitis received. In some embodiments, a local object may be first used inlocal storage for transaction N if the replication records exceed acache size threshold (not illustrated) before being spilled to remotedata object 534. On commit 532, transaction N remote data object 534 maybe inserted into replication log 550.

The database service and storage service discussed in FIGS. 2 through 5provide examples of a system that may perform amortizing replication logupdates for transactions. However, various other types of data stores(e.g., non-log structured) or other storage engines may implementamortizing replication log updates for transactions. FIG. 6 is ahigh-level flow chart illustrating methods and techniques for amortizingreplication log updates for transactions, according to some embodiments.Various different systems and devices may implement the various methodsand techniques described below, either singly or working together. Forexample, a database engine head node or storage node may implement thevarious methods. Alternatively, a combination of different systems anddevices. Therefore, the above examples and or any other systems ordevices referenced as performing the illustrated method, are notintended to be limiting as to other different components, modules,systems, or configurations of systems and devices.

As indicated at 610, receive a stream of updates to a database as partof an active transaction, in some embodiments. The updates may bereceived individually or in groups. The updates may describe changes todata values of records or items in the database, or may describe changesto the schema, settings, format, or other database information. Asindicated at 620, replication records for individual updates of thestream of updates may be generated as the updates are received. Forexample, logical replication records, such as those generated by MySQL'sBinlog feature, may be generated that describe the changes to beperformed to accomplish the updates, in some embodiments. In otherembodiments, physical replication records may be generated that includethe updated data values to be replicated.

As indicated 630, monitoring of the size of generated log records may beperformed. A transaction size threshold may be applied, which may, insome embodiments, compare the number of updates, number of replicationlog records, amount of change to the database, storage size ofreplication log records, length of time for performing the activetransaction, or any other measurement of transaction size. If thetransaction size threshold is not exceeded, then as indicated at 632 thereplication records of the stream of updates may be stored to a localdata sore. As discussed above with regard to FIGS. 4 and 5 , local datastores may include in-memory caches of replication log records or localtransaction data objects in persistent storage devices.

As indicated at 640, when a size of generated replication log recordsexceeds the transaction size threshold, then existing replicationrecords for the transaction in the local data store may be moved to atransaction data object in a remote data store. As indicated at 650, thenewly generated replication records for the stream of updates may thenbe stored to a transaction data object in a remote data store (after theexisting replication records are moved) instead of the local data store,in some embodiments. In some scenarios, a transaction data object may beused to store records until a size limit is reached (e.g., full), then anew transaction data object (which may be linked with or index to theprior transaction data object for the same transaction may be used).

FIG. 7 is a high-level flow chart illustrating methods and techniquesfor updating a replication log, according to some embodiments. Asindicated at 710, a commit request may be received for an activetransaction, in some embodiments. A determination may be made as towhether the transaction is using amortized replication for thereplication log, as indicated at 720. For example, a storage map, orother metadata may indicate whether, replication log records are storedlocally or remotely. In some embodiments, a transaction table mayindicate the type of replication.

If amortized replication is used, then a replication log for thedatabase may be updated to include replication records that describeupdates of the committed transaction in the data object, in someembodiments, as indicated at 730. For example, the transaction dataobject may be formatted like a large record in the log (or series of logrecords such as an append-only file), and thus may be included bymodifying pointers from previous log records in the replication log topoint forward to the transaction data object and subsequent log recordsto point back to the object. In this way, the replication log can beupdated without copying the records of the transaction data object orrecalculating log structure features, such as record offsets (which maybe zero as only a single transaction's records may be in the transactiondata object) or checksums (which may have been deferred until the end ofthe transaction is received). The transaction data object may be sortedinto the replication log according to a corresponding location for thecommit order of the transaction relative to other committedtransactions.

If a transaction is not amortized, then it may be committed with othertransactions or individually, in some embodiments, as discussed abovewith regard to FIG. 5 . For example, as indicated at 740, replicationrecords that describe updates of the committed transaction may beobtained from the local data store and stored into the replication logaccording to a transaction ordering for the replication log. Thesechange records may be written directly to the replication log.

The methods described herein may in various embodiments be implementedby any combination of hardware and software. For example, in oneembodiment, the methods may be implemented by a computer system (e.g., acomputer system as in FIG. 8 ) that includes one or more processorsexecuting program instructions stored on a computer-readable storagemedium coupled to the processors. The program instructions may beimplement the functionality described herein (e.g., the functionality ofvarious servers and other components that implement the databaseservices/systems and/or storage services/systems described herein). Thevarious methods as illustrated in the figures and described hereinrepresent example embodiments of methods. The order of any method may bechanged, and various elements may be added, reordered, combined,omitted, modified, etc.

FIG. 8 is a block diagram illustrating a computer system that mayimplement at least a portion of the systems and techniques foramortizing replication log updates for transactions described herein,according to various embodiments. For example, computer system 1000 mayimplement a database engine head node of a database tier, or one of aplurality of storage nodes of a separate distributed storage system thatstores databases and associated metadata on behalf of clients of thedatabase tier, in different embodiments. Computer system 1000 may be anyof various types of devices, including, but not limited to, a personalcomputer system, desktop computer, laptop or notebook computer,mainframe computer system, handheld computer, workstation, networkcomputer, a consumer device, application server, storage device,telephone, mobile telephone, or in general any type of computing device.

Computer system 1000 includes one or more processors 1010 (any of whichmay include multiple cores, which may be single or multi-threaded)coupled to a system memory 1020 via an input/output (I/O) interface1030. Computer system 1000 further includes a network interface 1040coupled to I/O interface 1030. In various embodiments, computer system1000 may be a uniprocessor system including one processor 1010, or amultiprocessor system including several processors 1010 (e.g., two,four, eight, or another suitable number). Processors 1010 may be anysuitable processors capable of executing instructions. For example, invarious embodiments, processors 1010 may be general-purpose or embeddedprocessors implementing any of a variety of instruction setarchitectures (ISAs), such as the x86, PowerPC, SPARC, or MIPS ISAs, orany other suitable ISA. In multiprocessor systems, each of processors1010 may commonly, but not necessarily, implement the same ISA. Thecomputer system 1000 also includes one or more network communicationdevices (e.g., network interface 1040) for communicating with othersystems and/or components over a communications network (e.g. Internet,LAN, etc.). For example, a client application executing on system 1000may use network interface 1040 to communicate with a server applicationexecuting on a single server or on a cluster of servers that implementone or more of the components of the database systems described herein.In another example, an instance of a server application executing oncomputer system 1000 may use network interface 1040 to communicate withother instances of the server application (or another serverapplication) that may be implemented on other computer systems (e.g.,computer systems 1090).

In the illustrated embodiment, computer system 1000 also includes one ormore persistent storage devices 1060 and/or one or more I/O devices1080. In various embodiments, persistent storage devices 1060 maycorrespond to disk drives, tape drives, solid state memory, other massstorage devices, or any other persistent storage device. Computer system1000 (or a distributed application or operating system operatingthereon) may store instructions and/or data in persistent storagedevices 660, as desired, and may retrieve the stored instruction and/ordata as needed. For example, in some embodiments, computer system 1000may host a storage node, and persistent storage 1060 may include theSSDs attached to that server node.

Computer system 1000 includes one or more system memories 1020 that maystore instructions and data accessible by processor(s) 1010. In variousembodiments, system memories 1020 may be implemented using any suitablememory technology, (e.g., one or more of cache, static random-accessmemory (SRAM), DRAM, RDRAM, EDO RAM, DDR 10 RAM, synchronous dynamic RAM(SDRAM), Rambus RAM, EEPROM, non-volatile/Flash-type memory, or anyother type of memory). System memory 1020 may contain programinstructions 1025 that are executable by processor(s) 1010 to implementthe methods and techniques described herein. In various embodiments,program instructions 1025 may be encoded in platform native binary, anyinterpreted language such as Java™ byte-code, or in any other languagesuch as C/C++, Java™, etc., or in any combination thereof. For example,in the illustrated embodiment, program instructions 1025 include programinstructions executable to implement the functionality of a databaseengine head node of a database tier, or one of a plurality of storagenodes of a separate distributed storage system that stores databases andassociated metadata on behalf of clients of the database tier, indifferent embodiments. In some embodiments, program instructions 1025may implement multiple separate clients, server nodes, and/or othercomponents.

In some embodiments, program instructions 1025 may include instructionsexecutable to implement an operating system (not shown), which may beany of various operating systems, such as UNIX, LINUX, Solaris™, MacOS™,Windows™, etc. Any or all of program instructions 1025 may be providedas a computer program product, or software, that may include anon-transitory computer-readable storage medium having stored thereoninstructions, which may be used to program a computer system (or otherelectronic devices) to perform a process according to variousembodiments. A non-transitory computer-readable storage medium mayinclude any mechanism for storing information in a form (e.g., software,processing application) readable by a machine (e.g., a computer).Generally speaking, a non-transitory computer-accessible medium mayinclude computer-readable storage media or memory media such as magneticor optical media, e.g., disk or DVD/CD-ROM coupled to computer system1000 via I/O interface 1030. A non-transitory computer-readable storagemedium may also include any volatile or non-volatile media such as RAM(e.g. SDRAM, DDR SDRAM, RDRAM, SRAM, etc.), ROM, etc., that may beincluded in some embodiments of computer system 1000 as system memory1020 or another type of memory. In other embodiments, programinstructions may be communicated using optical, acoustical or other formof propagated signal (e.g., carrier waves, infrared signals, digitalsignals, etc.) conveyed via a communication medium such as a networkand/or a wireless link, such as may be implemented via network interface1040.

In some embodiments, system memory 1020 may include data store 1045,which may be implemented as described herein. For example, theinformation described herein as being stored by the database tier (e.g.,on a database engine head node), such as a transaction log, an undo log,cached page data, or other information used in performing the functionsof the database tiers described herein may be stored in data store 1045or in another portion of system memory 1020 on one or more nodes, inpersistent storage 1060, and/or on one or more remote storage devices1070, at different times and in various embodiments. Similarly, theinformation described herein as being stored by the storage tier (e.g.,redo log records, coalesced data pages, and/or other information used inperforming the functions of the distributed storage systems describedherein) may be stored in data store 1045 or in another portion of systemmemory 1020 on one or more nodes, in persistent storage 1060, and/or onone or more remote storage devices 1070, at different times and invarious embodiments. In general, system memory 1020 (e.g., data store1045 within system memory 1020), persistent storage 1060, and/or remotestorage 1070 may store data blocks, replicas of data blocks, metadataassociated with data blocks and/or their state, database configurationinformation, and/or any other information usable in implementing themethods and techniques described herein.

In one embodiment, I/O interface 1030 may coordinate I/O traffic betweenprocessor 1010, system memory 1020 and any peripheral devices in thesystem, including through network interface 1040 or other peripheralinterfaces. In some embodiments, I/O interface 1030 may perform anynecessary protocol, timing or other data transformations to convert datasignals from one component (e.g., system memory 1020) into a formatsuitable for use by another component (e.g., processor 1010). In someembodiments, I/O interface 1030 may include support for devices attachedthrough various types of peripheral buses, such as a variant of thePeripheral Component Interconnect (PCI) bus standard or the UniversalSerial Bus (USB) standard, for example. In some embodiments, thefunction of I/O interface 1030 may be split into two or more separatecomponents, such as a north bridge and a south bridge, for example.Also, in some embodiments, some or all of the functionality of I/Ointerface 1030, such as an interface to system memory 1020, may beincorporated directly into processor 1010.

Network interface 1040 may allow data to be exchanged between computersystem 1000 and other devices attached to a network, such as othercomputer systems 1090 (which may implement one or more storage systemserver nodes, database engine head nodes, and/or clients of the databasesystems described herein), for example. In addition, network interface1040 may allow communication between computer system 1000 and variousI/O devices 1050 and/or remote storage 1070. Input/output devices 1050may, in some embodiments, include one or more display terminals,keyboards, keypads, touchpads, scanning devices, voice or opticalrecognition devices, or any other devices suitable for entering orretrieving data by one or more computer systems 1000. Multipleinput/output devices 1050 may be present in computer system 1000 or maybe distributed on various nodes of a distributed system that includescomputer system 1000. In some embodiments, similar input/output devicesmay be separate from computer system 1000 and may interact with one ormore nodes of a distributed system that includes computer system 1000through a wired or wireless connection, such as over network interface1040. Network interface 1040 may commonly support one or more wirelessnetworking protocols (e.g., Wi-Fi/IEEE 802.11, or another wirelessnetworking standard). However, in various embodiments, network interface1040 may support communication via any suitable wired or wirelessgeneral data networks, such as other types of Ethernet networks, forexample. Additionally, network interface 1040 may support communicationvia telecommunications/telephony networks such as analog voice networksor digital fiber communications networks, via storage area networks suchas Fibre Channel SANs, or via any other suitable type of network and/orprotocol. In various embodiments, computer system 1000 may include more,fewer, or different components than those illustrated in FIG. 8 (e.g.,displays, video cards, audio cards, peripheral devices, other networkinterfaces such as an ATM interface, an Ethernet interface, a FrameRelay interface, etc.)

It is noted that any of the distributed system embodiments describedherein, or any of their components, may be implemented as one or moreweb services. For example, a database engine head node within thedatabase tier of a database system may present database services and/orother types of data storage services that employ the distributed storagesystems described herein to clients as web services. In someembodiments, a web service may be implemented by a software and/orhardware system designed to support interoperable machine-to-machineinteraction over a network. A web service may have an interfacedescribed in a machine-processable format, such as the Web ServicesDescription Language (WSDL). Other systems may interact with the webservice in a manner prescribed by the description of the web service'sinterface. For example, the web service may define various operationsthat other systems may invoke, and may define a particular applicationprogramming interface (API) to which other systems may be expected toconform when requesting the various operations.

In various embodiments, a web service may be requested or invokedthrough the use of a message that includes parameters and/or dataassociated with the web services request. Such a message may beformatted according to a particular markup language such as ExtensibleMarkup Language (XML), and/or may be encapsulated using a protocol suchas Simple Object Access Protocol (SOAP). To perform a web servicesrequest, a web services client may assemble a message including therequest and convey the message to an addressable endpoint (e.g., aUniform Resource Locator (URL)) corresponding to the web service, usingan Internet-based application layer transfer protocol such as HypertextTransfer Protocol (HTTP).

In some embodiments, web services may be implemented usingRepresentational State Transfer (“RESTful”) techniques rather thanmessage-based techniques. For example, a web service implementedaccording to a RESTful technique may be invoked through parametersincluded within an HTTP method such as PUT, GET, or DELETE, rather thanencapsulated within a SOAP message.

The various methods as illustrated in the figures and described hereinrepresent example embodiments of methods. The methods may be implementedmanually, in software, in hardware, or in a combination thereof. Theorder of any method may be changed, and various elements may be added,reordered, combined, omitted, modified, etc.

Although the embodiments above have been described in considerabledetail, numerous variations and modifications may be made as wouldbecome apparent to those skilled in the art once the above disclosure isfully appreciated. It is intended that the following claims beinterpreted to embrace all such modifications and changes and,accordingly, the above description to be regarded in an illustrativerather than a restrictive sense.

What is claimed is:
 1. A system, comprising: at least one processor; anda memory, storing program instructions that when executed by the atleast one processor causing the at least one processor to implement adatabase; wherein the database is configured to: receive a request tobegin a transaction at the database; receive a stream of updates to thedatabase as part of the transaction; generate respective replicationrecords to be included to a replication log that describe individualupdates of the stream of updates as the updates are received; prior tocommitting the transaction to the database: store a portion of thegenerated replication records that describe the stream of updates in alocal cache of the database for the transaction in the memory; andresponsive to a determination that a size of the stored portion of thegenerated replication records exceeds a transaction size threshold,store the generated replication records into a transaction data objectin a remote data store instead of the local cache of the database,wherein the remote data store includes the replication log separatelyfrom the transaction data object; receive a request to commit thetransaction to the database; and update one or more pointers for one ormore other log records or other transaction data objects in thereplication log in the remote data store to point to the transactiondata object to include the respective replication records of thetransaction data object that were previously stored responsive to thedetermination that the size of the stored portion of the generatedreplication records exceeds the transaction size threshold withoutrewriting the respective replication records from the transaction dataobject into the replication log, wherein the replication records of thetransaction data object describe the individual updates of thetransaction that is committed to the database.
 2. The system of claim 1,wherein the database is further configured to: before the determinationthat the size of the stored portion of the generated replication recordsexceeds the transaction size threshold: determine that the size of thestored portion of the generated replication records exceeds a cache sizethreshold for the transaction; and responsive to the determination thatthe size of the stored portion of the generated replication recordsexceeds the cache size threshold for the transaction, store thegenerated replication records in a local transaction data object in alocal persistent storage device.
 3. The system of claim 1, wherein thesystem further comprises one or more persistent storage devices, andwherein the database is further configured to: receive a second streamof updates to the database for a second transaction; generatereplication records for the replication log that describe individualupdates of the second stream of updates as the second stream of updatesare received; store at least a portion of the replication records forindividual updates of the second stream of updates to a localtransaction data object in the one or more persistent storage devices;receive a request to commit the second transaction before determiningthat a size of the stored portion of the generated replication recordsfor the second stream of updates exceeds the transaction size threshold;and update the replication log for the database to include therespective replication records that describe the individual updates ofthe committed second transaction from the local transaction data object.4. The system of claim 1, wherein the at least one processor and thememory are implemented as part of a database service offered as part ofa provider network, wherein the remote data store is implemented as partof a storage service offered as part of the provider network, whereinreplication logging is enabled for the database responsive to a requestreceived via an interface for the database service.
 5. A method,comprising: receiving a stream of updates to a database as part of anactive transaction; generating respective replication records forindividual updates of the stream of updates as the updates are received;prior to committing the active transaction to the database: responsiveto determining that a size of the generated replication records exceedsa transaction size threshold, storing the generated replication recordsfor the stream of updates into a transaction data object in a remotedata store instead of to a local data store, wherein the remote datastore includes a replication log separately from the transaction dataobject; receiving a request to commit the active transaction to thedatabase; and updating one or more pointers for one or more other logrecords or other transaction data objects in the replication log in theremote data store to point to the transaction data object to include therespective replication records of the transaction data object that werepreviously stored responsive to the determination that the size of thestored portion of the generated replication records exceeds thetransaction size threshold without rewriting the respective replicationrecords from the transaction data object into the replication log,wherein the replication records of the transaction data object describethe individual updates of the active transaction that is committed tothe database.
 6. The method of claim 5, wherein the local data store isa local in-memory cache, and wherein the method further comprises:before determining that the size of the generated replication recordsexceeds the transaction size threshold, storing the generatedreplication records in the local in-memory cache.
 7. The method of claim5, wherein the local data store is a local persistent storage device,and wherein the method further comprises: before determining that thesize of the generated replication records exceeds a transaction sizethreshold, storing the generated replication records in the localtransaction data object in the local persistent storage device after thesize of the generated replication records exceeds a cache size thresholdfor storing the generated replication records in a local in-memorycache.
 8. The method of claim 5, further comprising: receiving a secondstream of updates to the database for a second active transaction;generating replication records for individual updates of the secondstream of updates as the second stream of updates are received;receiving a request to commit the second active transaction beforedetermining that a size of the generated replication records for thesecond stream of updates exceeds the transaction size threshold; andupdating the replication log for the database to include the respectivereplication records that describe the individual updates of thecommitted second transaction from a local transaction data objectstoring the generated replication records for the second stream ofupdates.
 9. The method of claim 8, wherein requests to commit one ormore additional active transactions are received, and wherein updatingthe replication log for the database to include the respectivereplication records that describe the individual updates of thecommitted second transaction from the local transaction data objectstoring the generated replication records for the second stream ofupdates comprises: inserting respective replication records for the oneor more additional active transactions from respective local transactiondata objects and the local transaction data object for the second activetransaction into the replication log for the database according to atransaction ordering.
 10. The method of claim 5, wherein the replicationlog orders transactions, including the active transaction, according toan order in which the transactions are committed, wherein the updatingthe replication log inserts the committed transactions into thereplication log according to the order.
 11. The method of claim 5,wherein updating the replication log for the database to include therespective replication records that describe the individual updates ofthe committed transaction in the transaction data object is performedwithout copying the respective replication records from the transactiondata object.
 12. The method of claim 5, wherein updating the replicationlog for the database to include the respective replication records thatdescribe the individual updates of the committed transaction in thetransaction data object is performed without duplicate calculation ofoffsets and checksums for the respective replication records.
 13. Themethod of claim 5, wherein the replication records are logicaldescriptions of the updates to the database.
 14. One or morenon-transitory, computer-readable storage media, storing programinstructions that when executed on or across one or more computingdevices cause the one or more computing devices to implement: receivinga stream of updates to a database as part of an active transaction;generating respective replication records for individual updates of thestream of updates as the updates are received; prior to committing theactive transaction to the database: storing the generated replicationrecords for the stream of updates in a local data store untildetermining that a size of the generated replication records exceeds atransaction size threshold; and after determining that the size of thegenerated replication records exceeds the transaction size threshold,storing the generated replication records into a transaction data objectin a remote data store, wherein the remote data store includes areplication log separately from the transaction data object; receiving arequest to commit the active transaction to the database; and updatingone or more pointers for one or more other log records or othertransaction data objects in the replication log in the remote data storeto point to the transaction data object to include the respectivereplication records of the transaction data object that were previouslystored responsive to the determination that the size of the storedportion of the generated replication records exceeds the transactionsize threshold without rewriting the respective replication records fromthe transaction data object into the replication log, wherein thereplication records of the transaction data object describe theindividual updates of the active transaction that is committed to thedatabase.
 15. The one or more non-transitory, computer-readable storagemedia of claim 14, wherein the local data store is a local persistentstorage device, and wherein the one or more non-transitory,computer-readable storage media store further program instructions thatwhen executed by the one or more computing devices cause the one or morecomputing devices to further implement: before determining that the sizeof the generated replication records exceeds a transaction sizethreshold, storing the generated replication records in the localtransaction data object in the local persistent storage device after thesize of the generated replication records exceeds a cache size thresholdfor storing the generated replication records in a local in-memorycache.
 16. The one or more non-transitory, computer-readable storagemedia of claim 14, further comprising: receiving a second stream ofupdates to the database for a second active transaction; generatingreplication records for individual updates of the second stream ofupdates as the second stream of updates are received; receiving arequest to commit the second active transaction before determining thata size of the generated replication records for the second stream ofupdates exceeds the transaction size threshold; and updating thereplication log for the database to include the respective replicationrecords that describe the individual updates of the committed secondtransaction from a local transaction data object storing the generatedreplication records for the second stream of updates.
 17. The one ormore non-transitory, computer-readable storage media of claim 16,wherein requests to commit one or more additional active transactionsare received, and wherein, in updating the replication log for thedatabase to include the respective replication records that describe theindividual updates of the committed second transaction from a localtransaction data object storing the generated replication records forthe second stream of updates, the program instructions cause the one ormore computing devices to implement: inserting respective replicationrecords for the one or more additional active transactions fromrespective local transaction data objects and the local transaction dataobject for the second active transaction into the replication log forthe database according to a transaction ordering.
 18. The one or morenon-transitory, computer-readable storage media of claim 14, whereinupdating the replication log for the database to include the respectivereplication records that describe the individual updates of thecommitted transaction in the transaction data object is performedwithout copying the respective replication records from the transactiondata object.
 19. The one or more non-transitory, computer-readablestorage media of claim 14, wherein the one or more non-transitory,computer-readable storage media store further program instructions thatwhen executed by the one or more computing devices cause the one or morecomputing devices to further implement: before beginning the activetransaction, receiving a request to enable replication logging inaccordance with the transaction size threshold.
 20. The one or morenon-transitory, computer-readable storage media of claim 14, wherein theone or more computing devices are implemented as part of a databaseservice that hosts the database in a provider network.